In these exercises, we will be going through the dataset you scraped and preprocess it for further analysis. If you couldn’t scrape your own dataset for some reason, you can use one provided by us: RawComments.Rds. Should you run into any problems, try using the clue boxes or asking us for help. For the solutions, we will be using the prepared dataset; if you are using your scraped one, be sure to replace the names and paths in the code accordingly.

Exercise 1

Load your scraped dataset or the one we prepared for you into your R-session and assign it to a dataframe called comments. Get an overview of the contained variables. What do the variables describe? Why do we have missing data in some of them?

To load the data, you can use the readRDS() function, to get an overview of the contained variables, you can simply use colnames(). To find out more about what the variables mean, you can go to the YouTube data API documentation and search for the comments ouput description.

Exercise 2

We want to remove the variables authorProfileImageUrl, authorChannelUrl, authorChannelUrl.value,video_id,canRate and viewerRating and moderationStatus. Create a new dataframe called Selection containing only the remaining variables.

You can use the subset() function to keep or remove a selection of variables from a dataframe. For more information, run ?subset()

Exercise 3

Check the class of the variable publishedAt in your new dataframe. Is this class suitable for further analysis? If not, change the class to the appropriate one and compute the time difference in publishing dates between the comment in the first row and the comment in the last row.

Do the same transformation for the variable updatedAt

To check the class of the publishedAt variable, you can use the class() function. To check the formatting of the comment timestamp, you can check the YouTube API documentation. To transfom character strings into datetime objects in R, you can use the base function as.POSIXct() or the more convenient anytime() function from the package with the same name.

Exercise 4

Check the likeCount variable in your data, is it suitable for numeric analysis? If not, transform it to the approrpiate class and test whether your transformation worked.

You can use the class() function to check the class of an R-object. To change a class, for example from character to numeric, you can use the family of “as”-functions, for example as.numeric()

Exercise 5

Check the textOriginal column in your Selection dataframe. There are still hyperlinks in the column that we should remove for later text analysis steps. Extract the hyperlinks from the textOriginal column into a new list called Links. In addition, create a new variable called LinksDel that contains the textOriginal but without the hyperlinks.

The qdabRegex package has many prebuild functions to detect, remove and replace specific character strings. You can for example use the rm_url() function to extract and replace hyperlinks. You can check it’s documentation with ?rm_url() to learn how to extract and how to replace hyperlinks.

Exercise 6

Check the LinksDel variable to see that there are still emoji contained in the column. For our later analysis, we want to do three things:

  1. Create one column without hyperlinks and emoji for easier text mining
  2. Create one column where emoji are replaced by a textual description for easier text mining
  3. Create one column containing only the textual description of emoji

To achieve this, we first need a dictionary of emoji and their corresponding textual descriptions in a usable format. Load the emo package and have a look at the contained dataframe jis. Copy it to a new dataframe called EmojiList. Afterwards, source the provided CamelCase.R script (in the scripts folder) to transform the textual description from regular case into CamelCase. Finally, create a new variable called TextEmoDel containing the text without the emoji (you can use the ji_replace_all() function from the emo package for that)

We provide you with a function that capitalizes the first character of each word. The function is called simpleCap() and the scripts name is CamelCase.R. You can load it into your workspace using the source() function and specifying it’s location. You can find the function in the scripts folder. Keep in mind that this function is only capitalizing the first letters of each word, you still need to get rid of the extra space characters. The gsub() function is a handy tool for this.

Exercise 7

Ultimately, we want to use our EmojiList dataframe to replace the instances of emoji in our text with the textual description. We can do that by looping through all emoji in all texts and replacing them one at a time. There is a problem however: Some emoji are made up of multiple “shorter” emoji. If we match part of a “longer” emoji and replace it with it’s text description, the rest will become unreadble. For this reason, we need to make sure that we replace the emoji from longest to shortest. Sort the EmojiList dataframe by the length of the emoji column from longest to shortest.

You can count the number of characters in a vector of text using the nchar() function. You can reorder dataframes using the order function and you can reverse an order using the rev() function.

Exercise 8

We now have a working dictionary for replacing emoji with a textual description! Create a new variable called TextEmoRep as a copy of the LinksDel variable. Next, loop through the ordered EmojiList and for every element in TextEmoRep, replace the contained emoji with “EMOJI_” followed by their textual description. You can use the rm_default() function from the qdapRegex package to replace custom patterns. Be sure to check the documentation so you can set the appropriate options for the function.

Beware: There will be warnings in your console even if you are doing everything right.

Loop through the dictionary sorted from longest to shortest emoji. You need to use a for loop to go through all emoji for all comments, one at a time. The paste() function is useful for adding the prefix “EMOJI_” in front of your textual descriptions. Don’t forget to set the arguments fixed = TRUE, clean = TRUE and trim = FALSE in your call to rm_default()

Exercise 8

We now have the original text column, and the text column with removed hyperlinks and were emoji are replaced with their textual descriptions (TextEmoRep). We need one more variable that only contains the textual desciptions of the emoji. You can use our predefined function ÈxtractEmoji() from the scripts folder to create this variable.

Use the source() function to source the ExtractEmoji.R script from the scripts folder and then sapply() the ExtractEmoji() function to the variable TextEmoRep. To remove useless rownames from the extracted Emojis, you can set names(Emoji) to NULL

Exercise 8

We now have selected all the variables we need, formatted them into the right formats, cleaned the text and extracted some additional information from it. Create a new dataframe called df that contains the following variables:

  • Selection$authorDisplayName

  • Selection$textOriginal

  • TextEmoRep

  • TextEmoDel

  • Emoji

  • Selection$likeCount

  • Links

  • Selection$publishedAt

  • Selection$updatedAt

  • Selection$parentId

  • Selection$id

Set the following names for the column in the new dataframe:

  • Author

  • Text

  • TextEmojiReplaced

  • TextEmojiDeleted

  • Emoji

  • LikeCount

  • URL

  • Published

  • Updated

  • ParentId

  • CommentID

Save the new dataframe as an RDS object with the name “ParsedComments.Rds”

You can use the cbind.data.frame() function to paste together multiple columns to a dataframe. You need to set the argument stringsAsFactors = FALSE though, to prevent strings from being interpreted as factor variables. In addition, the variables Links and Emoji are lists and can contain multiple values per row. For this reason, we need to enclose them with the I() function to be able to put them into a dataframe. You can save your result using the saveRDS() function.